Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: mysql generated schema syntax error #1845

Closed
wants to merge 8 commits into from

Conversation

waketzheng
Copy link
Contributor

@waketzheng waketzheng commented Jan 8, 2025

Description

Fixes #1836

Motivation and Context

  • Remove ' IF NOT EXISTS' for INDEX creation sql if dialect is mysql.
  • Delete extra space in CREATE INDEX when INDEX_TYPE is empty.

How Has This Been Tested?

make ci

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added the changelog accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

Copy link

codspeed-hq bot commented Jan 8, 2025

CodSpeed Performance Report

Merging #1845 will not alter performance

Comparing waketzheng:fix-mysql-schema (b0c4f8a) with develop (e34c2c0)

Summary

✅ 8 untouched benchmarks

@coveralls
Copy link

coveralls commented Jan 8, 2025

Pull Request Test Coverage Report for Build 12732215675

Details

  • 4 of 4 (100.0%) changed or added relevant lines in 3 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 90.252%

Totals Coverage Status
Change from base Build 12714096970: 0.0%
Covered Lines: 6432
Relevant Lines: 7011

💛 - Coveralls

if schema := generator.get_create_schema_sql(safe): # pragma: nobranch
if client.schema_generator.DIALECT == "mysql":
# MySQL does not support 'IF NOT EXISTS' syntax for `INDEX`
schema = schema.replace(" INDEX IF NOT EXISTS", " INDEX")
Copy link
Contributor

@henadzit henadzit Jan 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code is shared across all supported integrations, we shouldn't be really making hacks for particular databases here since we already use inheritance to customize the behavior of schema generation. Imagine if we start handling all edge cases for all databases here - the code will become unreadable pretty soon.

Looks like the issue is coming from https://github.com/tortoise/tortoise-orm/blob/develop/tortoise/indexes.py#L62. What is going to happen if we replace self.INDEX_CREATE_TEMPLATE with schema_generator.INDEX_CREATE_TEMPLATE?

I think it would be more correct to get rid of Index.get_sql altogether and just implement SQL generation for Index in schema_generator subclass, hence being able to customize it for each database.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@henadzit I add a class _db_mysql to the Index class.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@waketzheng what do you think about my other suggestions? We work quite a lot together on this project and it would help a lot if we communicate better. In this example it would help if you shared your thoughts on why you decided to approach this problem in a different way than you initially planned or than proposed by me.

I think it would be more correct to get rid of Index.get_sql altogether and just implement SQL generation for Index in schema_generator subclass, hence being able to customize it for each database.

What is going to happen if we replace self.INDEX_CREATE_TEMPLATE with schema_generator.INDEX_CREATE_TEMPLATE?

Copy link
Contributor Author

@waketzheng waketzheng Jan 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@henadzit I think word is limit. For me, code is much easy to understand than word. Many many times, I was able to get the point why sb code like that quickly without reading any comments.
This PR was expected to be a hotfix for the BUG. It may not be the best solution for the creation of Index. But keep the library work correctly is more important that make code better. I would rather fix bug soon and leave the refactor job later.
Why not replace self.INDEX_CREATE_TEMPLATE with schema_generator.INDEX_CREATE_TEMPLATE:
image
As the above picture showing, both tortoise/backends/mysql/schema_generator.py and tortoise.indexes.py have a INDEX_CREATE_TEMPLATE var, and they are different value. It mean that get rid of Index.get_sql altogether in schema_generator will cost much time to do the refactor. Let's keep small step forward, don't change too much in such a hotfix PR!

Copy link
Contributor

@henadzit henadzit Jan 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. It isn't a hot fix PR since the bug hasn't gotten into any release.
  2. I proposed the changes and you ignored my comments pretty much and made other changes. That isn't just about getting the point, it's simple politeness.
  3. if you look at any successful open-source projects, you will see a lot of discussions and good communication.
  4. Your changes at the moment aren't just a bug fix. You are introducing a new concept where the create index statements can be customized via class _db_*. The way it works in open-source and pretty much in any software development, other people will start following the pattern and extending the code base in a similar manner.
  5. We need to be trying to improve the code base with each our change, otherwise it will decay pretty quickly. "I will do a hot fix and then we refactor it later" rarely works.

I'll give a shot at this issue to see if there is an approach where we can simplify things.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@henadzit

  1. No matter whether it is a hot fix or not, we should keep the develop branch work fine.
  2. Actually, I try your suggestion get rid of Index.get in schema_generator and find it hard to do that, so I push commits with another solution, I expected to get your response after compare these two solutions.
  3. I read some PRs in poetry and fastapi, they did communicate a lot, but sometimes replied after several weeks. It is my pleasure that if we can communicate quickly and smoothly.
  4. class _db_* is not a new concept, you can find it at https://github.com/tortoise/tortoise-orm/blob/develop/tortoise/fields/data.py#L90
  5. Everyone have only 24 hours one day, we should focus on crucial things, "rarely" mean that there are jobs more important that this one.

I'll give a shot at this issue to see if there is an approach where we can simplify things.
OK, I'll pull your branch to local and add some tests.

@henadzit
Copy link
Contributor

henadzit commented Jan 10, 2025

Here's my shot at this issue , where I'm removing Index.sql and relaying only on schema_generator to build correct SQL. Please have a look!

@waketzheng
Copy link
Contributor Author

Here's my shot at this issue , where I'm removing Index.sql and relaying only on schema_generator to build correct SQL. Please have a look!

@henadzit My PR includes some useful tests. How about merge it and then your new PR can use them directly?



class TestDescribeModels(test.TestCase):
def test_describe_models_all_serializable(self):
val = Tortoise.describe_models()
json.dumps(val)
json.dumps(val, default=dump_index)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hides the issue that Index cannot be described in a serializable way, however, it doesn't fix it. I fixed it in my PR by implementing Index.describe.

@@ -150,6 +151,9 @@ class ModelTestPydanticMetaBackwardRelations3(Model):
class Node(Model):
name = fields.CharField(max_length=10)

class Meta:
indexes = [Index(fields=("name",))]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like you added this index because it exposes the issue with serializing Index in Model.describe. I was able to identify it only because I worked on the same issue in my PR, it wouldn't be clear to anyone reviewing this PR.

Even though this index exposes an issue, it isn't documented and completely not obvious why it is here. Someone can remove it later without knowing its purpose and the same regression can happen again. I added a dedicated class for testing indexes in my PR.

@waketzheng waketzheng closed this Jan 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

MySQL failed to create schema with indexes=[Index(fields=...)] in Meta
3 participants